In order to solve the problems of too many parameters and high computational complexity of traditional convolutional neural networks, a lightweight convolutional neural network architecture named C-Net based on cross-channel fusion and cross-module connection was proposed. Firstly, a method called cross-channel fusion was proposed. With it, the shortcoming of lacking information flow between different groups of grouped convolution was solved to a certain extent, and the information communication between different groups was realized efficiently and easily. Then, a method called cross-module connection was proposed. With it, the shortcoming that the basic building blocks in the traditional lightweight architecture were independent to each other was overcome, and the information fusion between different modules with the same resolution feature mapping within the same stage was achieved, enhancing the feature extraction capability. Finally, a novel lightweight convolutional neural network architecture C-Net was designed based on the two proposed methods. The accuracy of C-Net on the Food_101 dataset is 69.41%, and the accuracy of C-Net on the Caltech_256 dataset is 63.93%. Experimental results show that C-Net reduces the memory cost and computational complexity in comparison with the state-of-the-art lightweight convolutional neural network models. The ablation experiment verifies the effectiveness of the two proposed methods on the Cifar_10 dataset.